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(54) Method of processing Image Information and method of preventing forgery of certificates or 
the like 



(57) Additional information, composed of charac- 
ters, images, voice, etc.. is converted into two-dimen- 
sional codes (step S104) and then converted into a 
visible additional image to be embedded (step S105). 
The additional image is embedded in a full -color main 
image in a state of invisibility to produce a composite 
image (step S106). The composite image is recorded 
on a non-electronic medium such as paper or on an 
electronic medium, such as a memory on a personal 
computer, over the Internet (step S108). The embedded 
additional image is extracted from the composite image 
recorded on the recording medium and the additional 
information is reproduced (steps S109, S1 13). 
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based on two-dimensional codes; 

FIG. 4 is a diagram for use in explanation oi conversion of a to-be-embedded additional image into a bi-tevel image 
based on two-dimensional codes; 

FIG. 5 is a diagram for use in explanation of conversion of a to-be-embedded additional image into a bi-levei image 
5 based on two-dimensional codes; 

FIG 6 illustrates the procedure for creating a composite image; 

FIG. 7 illustrates an example of a specific computation in the composite image creation procedure; 

FIG 8 illustrates an example of a specific computation in the composite image creation procedure; 

FIG. 9 illustrates an example of a specific computation in the composite image creation procedure; 
io FIG. 1 0 illustrates an example of a specific computation in the composite image creation procedure; 

FIG 1 1 illustrates an example of a specific computation in the composite image creation procedure; 

FIG 12 illustrates an example of a specific computation in the composite image creation procedure; 

FIG. 13 illustrates an example of a specific computation in the composite Image creation procedure; 

FIG. 14 illustrates an example of a specific computation in the composite image creation procedure; 
is FIG. 15 illustrates an example of embedding information in a landscape photograph; 

FIG. 1 6 is a diagram for use in explanation of alignment marks in a composite image; 

FIG. 1 7 shows an example of a bt-level mask image; 

FIG. 18 is a block diagram of an image creating and recording system; 

FIG. 19 is a block diagram of an image reproducing system; 
20 FIG. 20 is a flowchart illustrating an image information processing method according to a first embodiment of the 

present invention; 

FIG. 21 shows an example of an identity card for explaining a second embodiment of the present invention; 
FIG. 22 is a diagram for use in explanation of printing (recording) a face photograph onto an identity card according 
to the second embodiment of the present invention; and 
25 FIGS. 23A, 23B and 23C each show a relationship between the amount of color difference and data values of a 
main image in which additional information is to be embedded. 

[0016] Referring now to FIG. 1. there is shown a flowchart illustrating an image information processing method 
according to the present invention. Hereinafter, the image information processing method of the present invention will 

so be described with reference to this flowchart 

[0017]. Additional information 101 to be embedded includes characters, images, sound, etc., which are information a 
creator wants to embed invisibly. A main image 1 02 is an image in which additional information is to be embedded, such 
as a photograph of a person's face on an identity card. The main image serves as information for producing a full-color 
image. A mask image (pattern image information) 103 is two-valued image information which is used in a combining 

35 process (step Si 06) and a recovery process (a) or (b) (step S1 09 or S11 2) as will be described later. 

[001 8] In the first place, in binary coding step S1 04, the to-be-embedded information 1 01 is binary-coded (two-valued 
and coded). In this case, analog data, such as sound, is first subjected to analog-to-digrtal conversion and then binary- 
coded. On the other hand, digital data, such as characters and images, is binary-coded as it is. 
[0019] Next, in to-be-embedded image creating step S105. the binary (two-valued) data produced in binary coding 

40 step Si 04 is converted into binary image data in accordance with a predetermined format thereby producing a to-be- 
embedded additional image. By this process, even data that is usually invisible, such as sound data, is converted into 
visible data (binary image data). 

[0020] Next, in combining step S106, the to-be-embedded image produced in step S10S. the main image 102, and 
the mask image 103 are combined into a full-color composite image 107. The composite image 107, when made visi- 
45 We. apparently looks identical to the main image 102; the additional image is hidden in a state of invisibirrty. The com- 
posite image 107 can be stored in a commonly used format, for example, TIFF or JPEG. 

[0021] The composite image 107 thus produced can be distributed via ether an electronic medium, such as a home 
page on the Internet or electronic mail, or a non-electronic medium such as paper. 

[0022] First the distribution of the composite image via an electronic medium will be described. In recording step 
so S108. the composite image 107 produced in combining step S106 is recorded on an electronic medium, such as a 
home page or electronic mail on the Internet 

[0023] Next, a person who received the composite image recorded on the electronic medium performs a recovery 
operation in step (a) S109 to make visible the additional image invisibly embedded in the composite image. After that, 
the recipient performs a reproduction operation in step (a) Si 1 0 to reproduce the embedded additional information 101 . 
55 [0024] The distribution of the composite image through a non-electronic medium will be described. In recording step 
S1 1 1. the composite image 107 produced in combining step Si 06 is recorded on a non-electronic medium such as 
paper. 

[0025] Next, a person who received the composite image 107 recorded on the non-electronic medium performs a 
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Then, these are replaced with image elements as shown in FIG. 4. 

[0032] Further, the bilevel image information is enlarged by a factor of n in order to prevent the degradation of to-be- 
5 embedded images during a smoothing step in the combining process to be described later. Here, it is preferable that n 
= 2, 3 or 4. In FIG. 5 there is shown the result of enlarging the bteveJ image information of FIG. 4 by a factor of two. 
[0033] Although, in this embodiment, the calra code is applied to the two-dimensional coding, a matrix type two* 
dimensional code or a two-dimensional bar code, such as the glyph code, can be used. 

[0034] . After the replacement of all the to-be-embedded additional information with two-dimensional codes, in step 
io St 05. these are laid out two-dimensionally to obtain the to-be-embedded images. In this example, as the to-be-embed- 
ded image, the header a, the first information b, the second information c. the third information d, and the fourth infor- 
mation e are converted into two-dimensional codes. 

[0035] At the time of creation of the to-be-embedded information header and the to-be-embedded information, each 
piece of to-be-embedded information Is related with the main image. This provides a criterion for allowing users to draw 

is certain specific information from multiple pieces of information embedded in a composite image. 

[0036] For example, in a case where the present invention is applied to an electronic picture book, suppose that pho- 
tographs of a zoo are used as main images. Among these photographs, for example, a photograph of Sons will be asso- 
ciated with their cries and a photograph of penguins will be associated with their ecology. Thus, if a main image and 
embedded information are related so that the main image can be relatively easily associated with the embedded intor- 

20 mation, then it will become easy for users to find desired one from multiple pieces of embedded information. 

[0037] Furthermore, tor example, when the present Invention is applied to an identity card, a photograph of the 
owner's face is used as a main image and personal information that identifies the owner is used as to-be-embedded 
information The personal information may be made of fingerprints or voiceprint Voiceprtnt information can be embed- 
ded in a portion of the face photograph that corresponds with the mouth. 

25 [0038] Next, the combining step S106 of FIG. 1 will be described in more detail with reference to FIG. 6. 

[0039] The main image 102 is image information in which additional information is to be embedded and. in the case 
of an identity card, corresponds to an owner's face photograph. The image information has 24 bits of information per 
pixel (8 bits tor each of R. G and B components). The to-be-embedded image 105 is a bilevel image obtained by con- 
verting additional information through the previously described technique and. in the case of an identity card, corre- 

30 sponds to an identification number by way of example. This information has one bit of information per pixel. The mask 
image 103 is imagB information used in the combining process and the embedded image recovery process and has one 
bit of information per pixel. 

[0040] . First a smoothing process is performed In step S301 with black pixels in the to-be-embedded image 105 as 
is and white pixels as 0s. Here, a region of 3 x 1 pixels composed of a pixel of interest and two pixels on both sides of 
35 that pixel in the x direction is cut out, and the weighted mean is taken as follows: 

W(i) m (STLO-1) + 2 • STL® + STL(i+1))/4 (1) 

where 

40 

W(i) = the weighted mean of the Mh pixel 

STLfl) e embedded image at the i-th pixel (1 or 0). 

[0041] Attention must be given to the fact that, unless the to-be-embedded image is enlarged by a factor of n at the 
45 creation time as descrtoed in connection with FIG. 5, it is destroyed during the smoothing process. The larger the 
enlargement factor, the more the factor of safety of to-be-embedded image increases. However, as the enlargement 
factor increases, information that should be hidden becomes easier to be revealed. 

[0042] For example, when a to-be-embedded additional image is as shown in FIGL 8, the results of the smoothing 
process will be as shown in FIG. 9. In these figures, the enlargement factor n is set to n • 4, so that the image is 
so enlarged by a factor of 4 in the vertical and horizontal dimensions. To leave a margin for embedding, each pair of two 
adjacent pixels on the periphery is set to 0s as shown in FIG. 8. 

[0043] Next, in phase modulation step S3 02. on the basis of the results of the smoothing process in step S301 . phase 
modulation is performed on the mask image 1 03 in accordance with the following rules: 

55 lfW(i) = 0.thenDES(i) = MSK(i) (2-1) 

If W(i) = 1. then DES(i) ° MSK(k2) (2-2) 



CIO: «EP 0821676A2J_» 



5 



EP 0 921 675 A2 



[0049] The results tor red component when all the pixel values of the main image 102 are 127, i.e., when (R, G, B) o 
(127. 127, 127). are shown in FIG. 12. Each pixel value takes an integer in the range of 0 to 255. The red conrponent 
has the maximum value of 255. In this figure, the (0> 0) pixel and the (1 , 0) pixel each have a value of 79. and the (2. 0) 
pi iei and the (3. 0) pixel each have a value of 1 75. That is. in an area where no additional image is embedded, two sue- 
5 cessive pixels in which there is little red component and two successive pixels in which the red component is rich alter- 
nate 

[0050] As can be seen from equations (3-1) to (3-3) or (3-4) to (3-6). the color difference amount for red component 
is opposite in sign to those for green and blue components. Therefore, in a pixel in which the red component prevails, 
the green and blue components are reduced. In a pixel in which the red component is reduced, on the other hand, the 
to otnet coky components are increased. Green plus blue (i.e., cyan) is the complement of red. Thus, even if red and cyan 
are located adjacent to each other they are difficult to recognize by human eyes and looks achromatic' Since red-rich 
piers and cyan-rich pixels are alternated in units of two pixels, the human visual system cannot recognize the differ* 
ence m color between pixels and hence will determine that the color difference amount is 0. 
[005 1 ] In tefms of the human visual system, for example, equation (4-1) will be mistakenly taken to be 

13 

DESR(0»SRCR(i) (5) 

|0052] That «t humans cannot perceive the fact that additional image information has been embedded. H is on this 
pnnopie thai a composite image can be produced in which additional image information is embedded in a main image 
so in a ctate & *vtt«*lity 

[0053) A composite image <s usually recorded on an electronic medium in a general-purpose imaging format such 
as JPEG. TIFF, or the I fee However, in this embodiment since to-be-embedded information does not depend on any 
imaging Id* mat not only presently available imaging formats but also formats that win be developed in future can be 
employed 

25 [0054] A method of recovering embedded information from a composite image and reproducing it will be described 
hereinafter 

[0055] m the case of cfcstrtx/tion via an electronic medium such as a home page on the Internet or electronic mail, 
the contents of a composite image are recorded on the electronic medium in such a form as shown in FIG. 12. To 
recover thu mage, the mask image 103 shown in FIG. 7 is used. The pixels of the mask image 103 are made to corre- 
30 spond one for one with the pixels of the composite image 107. The composite image is made effective in areas in which 
the mask image 103 takes a value of 1 or ineffective in areas in which the mask image pixels have a value of 0. The 
results are shown in FIG. 13. In this figure, shaded pixels are ineffective ones. The effective data pixels (shown 
unshaded) are cut out in a predetermined size. 

[0056] m FIG a. the enlargement factor, n, is set such that n = 4 and the embedded information is quadrupled. After 

35 the removal of pixels on the periphery provided for a margin for embedding, effective data is cut out in units of 4 x 4 
pixels If aO the effective data pixels in a 4 x 4 pixel area take a red-rich value (1 75 in this example), then the embedded 
image in this area rs made 1 . If, on the other hand, they take a cyan-rich value (79 in this example), then the embedded 
image is made 0. When the effective data area contains both red-rich pixels and cyan-rich pixels, the embedded image 
depends on the pixels which are larger in number. For example, if the red-rich pixels are larger in number than the cyan- 

40 rich pixels, then the embedded image is made 1 ; otherwise, the embedded image is made 0. That both the red-rich pix- 
els and the cyan-rich pixels are contained is attributed to the smoothing step in the combining process. 
[0057) The lesurts of recovery of the embedded image by this method are shown in FIG. 14. In this figure, portions 
enclosed by bold fanes corresponds to the embedded image and matches with that of FIG. 8. It thus will be seen that 
the embedded information can be recovered completely. Furthermore, by reversing the procedure of creating the 

as embedded image described in conjunction with FIG. 2, the embedded information can be reproduced. 

[0058] When a composite image is printed on a non-electronic medium, such as paper, with a color printer, the same 
method can be used to recover an embedded image. In this case, the non-electronic medium printed with the compos- 
ite image is read by an optical reader and digitization is then performed to obtain the state of FIG. 12. After that the 
embedded information is simply recovered by the same method as described above. 

so [0059] After printing at a resolution of 400 dpi by means of a thermal printer in accordance with this embodiment and 
reading at a resolution of 1200 dpi by means of an optical scanner, recovery was made successfully. 
[0060] In order to reproduce additional information from a non-electronic medium, another method of reproduction 
can be used by which a mask sheet is physically superimposed on the composite image, which has the transmission 
factor distribution of the same pattern as the mask image 103 and is produced at the same recording resolution as that 

55 at the time of recording a composite image. This method has an advantage of requiring no troublesome operations and 
complex equipment in recovering embedded information. 

[0061 ] There is shown in FIG. 15 an embodiment in which copyright information and narration and associated back- 
ground music (BGM) are embedded in, for example, a landscape photograph as the main image 102. A to-oe-embed- 
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the embedded image to be identified visually. 

[0073] Next, a description will be given of an image information processing device tor implementing the image infor- 
mation processing method described so far. The processing device is constructed roughly from an image creating and 
recording device which creates and records a composite image on either an electronic medium or a non-electronic 
£ medium, and an image reproducing device which reproduces an embedded image from the composite image recorded 
on the electronic or non-electronic medum. 

[0074] First, the image creating and recordng device will be described with reference to RGL 18. A CPU (centra) 
processing unit) 401 performs the overall control of the device. A ROM (read-only memory) 402 stores programs that 
the CPU 401 executes to control the operation of the entire device. A RAM (random access memory) 403 serves as a 
10 working memory of the CPU 401 . An image memory 404 stores image information when images are combined and is 
used when processes are performed which will be described later. 

[0075] A to-be-embedded image is divided into fixed information and fluid information. The fixed information includes 
serial numbers inherent in a system and so on, whereas the fluid information includes the date and time of processing, 
identification numbers, voice, etc The fixed information and the fluid information are entered from a to-be-embedded 
i5 image ROM 409 and a to-be-embedded image input unit 410, respectively, into a to-be-embedded image creation unit 
408. A to-be-embedded image is created in accordance with the procedure shown in FIG. 2 and then stored in the 
image memory 404. 

[0076] A main image 1 02. such as a photograph of the face of a person or a landscape photograph, is entered from 
a main image input unit 407 into a separate area of the image memory 404. 

so [0077] As instructed by an operator, the CPU 401 operates a smoothing unit 41 1 to perform a smoothing process on 
the to-be-embedded image stored in the image memory 404 and then sends the results to a phase modulation unit 41 2. 
[0078] The phase modulation unit 41 2 performs a phase modulation process as shown in FIG. 1 1 on the basis of the 
results of the smoothing process and a mask image 1 03 from a mask image ROM 405 and then sends the results to a 
color difference modulation unit 413. 

25 [0079] The color difference modulation unit 41 3 performs a color difference modulation process as shown in FIG. 1 1 
on the basis of the results of the phase modulation process and color difference amount information from a color differ- 
ence amount ROM 406 and then passes the results to a superimposrfon processing unit 414. 
[0000] The superimpositon processing unit 41 4 performs a superimpose on process on the results of the color differ- 
ence modulation and the main image 102 stored in the image memory 404 to produce a composite image 107 and 

30 stores it into the image memory 404. 

[0081] When the composite image 107 is distributed via an electronic medium, H is distributed from a composite 
image output unit (electronic medium) 415 via a network by way of example. In the case of distribution via a non-elec- 
tronic medium, the composite image is output from a composite image output unit (non -electronic medium) 416 com- 
posed of a printer driver and a color printer. That is, the composite image 417 (107) is output in printed form. 

3s [0082] -The image reproducing device will be described next with reference to FIG. 19. A CPU 501 performs the overall 
control of the device. A ROM 502 stores programs that the CPU 501 carries out to control of the operation of the entire 
device. A RAM 503 is used as a working memory of the CPU 501. An image memory 504 stores image information., 
when embedded information is recovered and is used tor processes which will be described later. 
[0083] A composite image 507 (107) recorded on a non-electronic medium, such as paper, is read by a composite 

40 image reader 506 and then converted into digital data, which is in turn stored in an image memory 504. The composite 
image reader includes an optical scanner and its driver. 

[0084] A composite image recorded on an electronic medium Is directly stored Into the Image memory 504. The align- 
ment marks M as shown in FIQ. 16 are read out at the same time the composite image 507 is read and then stored in 
the RAM 503. 

45 [0085] As instructed by the operator, the CPU 501 controls each unit to recover the embedded information. 

[0086] The mask image creation unit 508 identifies the type of the mask image from the alignment mark information 
stored in the RAM 503. takes out of the mask image ROM 505 a mask information necessary for recovery and repro- 
duction of the embedded image, creates mask image, and stores tl into the image memory 504. 
[0087] An embedded image visualization unit 509 for mating visible the embedded image superimposes the mask 

so image 103 and the composite image 507 (107) upon each other as illustrated in FIGS. 7 and 8 and extracts only the 
effective data of the embedded image, thereby visualizing the embedded imaga After that an embedded image repro- 
duction unit 51 0 performs the procedure which is the reverse of that of FIQ. 2. thereby reproducing the embedded infor- 
mation. 

[0088] As described so far. according to this embodiment a composite image in which additional information is super- 
55 imposed on a main image can be created irrespective of which of electronic and non-electronic media it is to be 
recorded on. In addition, irrespective of the type of a recording medium used, the composite image can be recorded 
and added image information can be reproduced. 

[0089] Therefore, multimedia contents, such as electronic picture books, picture mail, sounding pictures, and so on. 
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degrades as the security strength increases. 

[01 06] In the third embodiment by sharing a role between the lace photograph image for identifying the person him- 
self or herself and the face photograph image for security, the image quality and the security strength can be made com- 
patible with each other. 

5 [01 07] The photograph image 5 on the last page 3 appears to human eyes as a black and white image: however, since 
it is composed of three primary colors of R. G. and B (or C. M, and Y). it, like the photographic image 4 on the first page, 
can be printed out by a usual color printer. That is. the apparatus can be used in common, which is very economical. 
[0108] Although, in the third embodiment two types of images, the identification image and the security image, are 
used, they do not necessarily need to be printed on the first page and the last page H. in the identity card, there exists 

to a page which is very secure, tor example, a watermarked page, the security image can be printed on that page to fur- 
ther increase the security. 

[01 09] In the third embotiment. which uses the image information processing method of the second embodiment, the 
security strength and the image quality are not compatible with each other as descrfoed previously. Thus, the Image 
quality degrades as the security strength increases. Hereinafter, causes of the degradation of image quality wifl be cfls- 
ts cussed. 

[01 10] As described previously, composite image data are expressed by 



DESR(i) o VR(i) ♦ SRCR(i) (4-1) 
so DESG(i) - VG(i) + SRCGffl (4-2) 

DESB(i) « VB(i) + SRCBffl (4-3) 



where 

23 

DESR(i) a result of superimposition at the i-th pixel for the red component (an integer in the range of 0 to 255) 
DESGfi) o result of superimposition at the i-th pixel for the green component (an integer in the range of 0 to 255) 
OESB(i) » result of superimposition at the i-th pixel for the blue component (an integer in the range of 0 to 255) 
SRCR(i) « main image at the i-th pixel for the red component (an integer in the range of 0 to 255) 
30 SRCG(i) » main image at the i-th pixel for the peen component (an integer in the range of 0 to 255) 
SRCB(i) a main image at the i-th pixel for the blue component (an integer in the range of 0 to 255). 

[01 1 1] Note that DESR(i), DESGfi) and DESB(i) each take an integer in the range of 0 to 255. For example, when 
VR(i) a 48 and SRCR(i) = 240. DESRfO =48 + 240 ° 288, exceeding 255. In practice, when 255 is exceeded, an over- 
35 flow results, so that 38 by which the result exceeds 255 is thrown away. Properly speaking, with DESR(i) « 286. the 
embedded image should have well balanced with the composite image, so that it is placed in a state of invisibility. How- 
ever, since DESR(i) ts4imited to 255 as a result of overflow, the embedded image (security information) becomes visi- 
ble. 

[01 1 2] This is the cause of the degradation of quality of the composite image. In order to minimize the degradation, it 
40 is required to decrease the color difference amount AV so that the overflow is minimized. However, the security strength 
lowers as AV decreases. Although the case of overflow has been described so far, the same phenomenon also occurs 
with underflow in which DESR(i) Is less than 0. 

[01 13] In the previously-described image information processing method of the second embodiment the color differ- 
ence amount AV is fixed at vb regardless of SRC (indicating data values of the embedded image) shown on the vertical 

45 axis as shown in FIG. 23 A. 

[01 1 4] By establishing a trapezoid relationship between AV and SRC as shown in FIG. 23B. that is, by making AV vary 
with the embedded image data value in a region close to 0 where underflow is likely to occur and in a region close to 
255 where overflow is likely to occur, underflow or overflow is made difficult to occur. This improves balance between 
the composite image and the embedded image, allowing the image quality of the composite image to be improved with 

so the security strength maintained. 

[01 1 5] In this case as well, however, there arises the problem that, when the main image data value is just 0 or 255. 
no embedding is performed. 

[01 1 6] These problems can be circumvented by setting a minimum color difference amount Vmin as shown in FIG. 
23C. Strictly speaking, overflow or underflow will occur when the color difference amount is at Vmin. Thus, the image 
55 quality degrades in comparison with the case of FIG. 23A; however, this level of degradation is out of the question as 
compared with the case of FIG. 23A. By setting Vmin to Vo/2 on an experience basts, the image quality of the composite 
image and the strength of security can be made compatible with each other. 

[01 1 7) As described so far, according to the second and third embodiments of the present invention, an image infer- 
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and its respective pixel is composed of multiple color components: and 
recording the composite image on a recording medium. 
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